Error correction in speech recognition

ABSTRACT

New techniques and systems may be implemented to improve error correction in speech recognition. These new techniques and systems may be implemented to correct errors in speech recognition systems may be used in a standard desktop environment, in a mobile environment, or in any other type of environment that can receive and/or present recognized speech.

CROSS REFERENCE TO RELATED APPLICATION

This application claims benefit to U.S. Application No. 60/201,257,filed May 2, 2000, which is incorporated herein by reference.

TECHNICAL FIELD

This invention relates to error correction in computer-implementedspeech recognition.

BACKGROUND

A speech recognition system analyzes a user's speech to determine whatthe user said. Most speech recognition systems are frame-based. In aframe-based system, a processor divides a signal descriptive of thespeech to be recognized into a series of digital frames, each of whichcorresponds to a small time increment of the speech.

A continuous speech recognition system can recognize spoken words orphrases regardless of whether the user pauses between them. By contrast,a discrete speech recognition system recognizes discrete words orphrases and requires the user to pause briefly after each discrete wordor phrase. Continuous speech recognition systems typically have a higherincidence of recognition errors in comparison to discrete recognitionsystems due to complexities of recognizing continuous speech.

In general, the processor of a continuous speech recognition systemanalyzes “utterances” of speech. An utterance includes a variable numberof frames and may correspond to a period of speech followed by a pauseof at least a predetermined duration.

The processor determines what the user said by finding acoustic modelsthat best match the digital frames of an utterance, and identifying textthat corresponds to those acoustic models. An acoustic model maycorrespond to a word, phrase or command from a vocabulary. An acousticmodel also may represent a sound, or phoneme, that corresponds to aportion of a word. Collectively, the constituent phonemes for a wordrepresent the phonetic spelling of the word. Acoustic models also mayrepresent silence and various types of environmental noise.

The words or phrases corresponding to the best matching acoustic modelsare referred to as recognition candidates. The processor may produce asingle recognition candidate (that is, a single sequence of words orphrases) for an utterance, or may produce a list of recognitioncandidates.

Correction mechanisms for some discrete speech recognition systemsdisplayed a list of choices for each recognized word and permitted auser to correct a misrecognition by selecting a word from the list ortyping the correct word. For example, DragonDictate™ for Windows™, byDragon Systems, Inc. of Newton, Mass., displayed a list of numberedrecognition candidates (“a choice list”) for each word spoken by theuser, and inserted the best-scoring recognition candidate into the textbeing dictated by the user. If the best-scoring recognition candidatewas incorrect, the user could select a recognition candidate from thechoice list by saying “choose-N”, where “N” was the number associatedwith the correct candidate. If the correct word was not on the choicelist, the user could refine the list, either by typing in the first fewletters of the correct word, or by speaking words (for example, “alpha”,“bravo”) associated with the first few letters. The user also coulddiscard the incorrect recognition result by saying “scratch that”.

Dictating a new word implied acceptance of the previous recognition. Ifthe user noticed a recognition error after dictating additional words,the user could say “Oops”, which would bring up a numbered list ofpreviously-recognized words. The user could then choose apreviously-recognized word by saying “word-N”, where “N” was a numberassociated with the word. The system would respond by displaying achoice list associated with the selected word and permitting the user tocorrect the word as described above.

SUMMARY

New techniques and systems improve error correction in speechrecognition. These techniques and systems may be used in a standarddesktop environment, in a mobile environment, or in any other type ofenvironment that can receive and/or present recognized speech. Moreover,the techniques and systems also may leverage the power of continuousspeech recognition systems, such as Dragon NaturallySpeaking,™ availablefrom Dragon Systems, Inc. of Newton, Mass., the capabilities of digitalrecorders and hand-held electronic devices, and the advantages of usinga contact manager or similar system for personal information management.

In one general aspect, a method of correcting incorrect text associatedwith recognition errors in computer-implemented speech recognitionincludes performing speech recognition on an utterance to produce arecognition result for the utterance and receiving a selection of a wordfrom the recognized utterance. The selection indicates a bound of aportion of the recognized utterance to be corrected. A first recognitioncorrection is produced based on a comparison between a first alternativetranscript and the recognized utterance to be corrected. A secondrecognition correction is produced based on a comparison between asecond alternative transcript and the recognized utterance to becorrected. A portion of the recognition result is replaced with one ofthe first recognition correction and the second recognition correction.A duration of the first recognition correction differs from a durationof the second recognition correction. Furthermore, the portion of therecognition result replaced includes at one bound the word indicated bythe selection and extends for the duration of the one of the firstrecognition correction and the second recognition correction with whichthe portion is replaced.

Implementations may include one or more of the following features. Forexample, the selection may indicate a beginning bound or a finishingbound of a recognized utterance to be corrected.

The comparison between an alternative transcript and the recognizedutterance may include selecting from the alternative transcript a testword that is not identical to the selected word. The test word begins ata time that is nearest a time at which the selected word begins. Thecomparison between the alternative transcript and the recognizedutterance may further include searching in time through the recognizedutterance and relative to the selected word and through the alternativetranscript and relative to the test word until a word common to therecognized utterance and the alternative transcript is found. The commonword may begin at a time in the recognized utterance that isapproximately near a time at which the common word begins in thealternative transcript.

Production of a recognition correction may include selecting a wordstring from the alternative transcript. The word string is bound by thetest word from the alternative transcript and by a word from thealternative transcript that is adjacent to the common word and betweenthe test word and the common word. The method may include receiving aselection of one of the first recognition correction and the secondrecognition correction.

Searching in time through the recognized utterance and through thealternative transcript may include designating a word adjacent to thetest word as an alternative transcript word, designating a word adjacentto the selected word as an original transcript word, and comparing theoriginal transcript word to the alternative transcript word.

The original transcript word and the alternative transcript word may bedesignated as the common word if the original transcript word isidentical to the alternative transcript word and if a time at which theoriginal transcript word begins is near a time at which the alternativetranscript word begins.

A word in the alternative transcript that is adjacent to the alternativetranscript word may be designated as the alternative transcript wordwhether or not the original transcript word is identical to thealternative transcript word if the original transcript word begins at atime that is later than a time at which the alternative transcript wordbegins. A word in the original transcript that is adjacent to theoriginal transcript word may be designated as the original transcriptword whether or not the original transcript word is identical to thealternative transcript word if the original transcript word begins at atime that is earlier than a time at which the alternative transcriptword begins. A word in the original transcript that is adjacent to theoriginal transcript word may be designated as the original transcriptword and a word in the alternative transcript that is adjacent to thealternative transcript word may be designated as the alternativetranscript word if the original transcript word is not identical to thealternative transcript word and if a time at which the originaltranscript word begins is near a time at which the alternativetranscript word begins.

A floating-choice-list system provides an advantage over priorchoice-list systems when used in hand-held or portable devices, whichoften require use of a stylus as an input device. In such a stylussystem, it would be difficult for a user to select two or more words tobe corrected using prior choice-list systems. In particular, users wouldbe required to perform the difficult task of carefully selecting a rangeof words to be corrected using a stylus before selecting an alternativetranscript. The floating-choice-list system simplifies the requiredstylus events needed to perform a multiword correction for speechrecognition on a hand-held device. Using the floating-choice-listsystem, the user only needs to contact the stylus somewhere in the wordthat begins the error-filled region in order to obtain a list ofalternative transcripts.

In another general aspect, a method of correcting incorrect textassociated with recognition errors in computer-implemented speechrecognition includes receiving a text document formed by recognizingspeech utterances using a vocabulary. The method also includes receivinga general confusability matrix and receiving corrected text. The generalconfusability matrix has one or more values, each value indicating alikelihood of confusion between a first phoneme and a second phoneme.The corrected text corresponds to misrecognized text from the textdocument. If the corrected text is not in the vocabulary, the methodincludes generating a sequence of phonemes for the corrected text. Thegenerated sequence of phonemes is aligned with phonemes of themisrecognized text and one or more values of the general confusabilitymatrix are adjusted based on the alignment to form a specificconfusability matrix. The method further includes searching the textdocument for additional instances of the corrected text using thespecific confusability matrix.

Implementations may include one or more of the following features. Themethod may further include outputting the text document. A list ofrecognition candidates may be associated with each recognized speechutterance. The step of generating the sequence of phonemes for thecorrected text may include using a phonetic alphabet.

The method may also include generating the general confusability matrixusing empirical data. In that case, the empirical data may includeinformation relating to a rate of confusion of phonemes for apreselected population, information relating to frequencycharacteristics of different phonemes, or information acquired during anadaptive training of a user.

The step of searching the text document for the corrected text mayinclude searching the text document for the sequence of phonemes for thecorrected text. The step of searching the text document for thecorrected text may include searching the text document for a sequence ofphonemes that is likely to be confused with the sequence of phonemes forthe corrected text.

The step of searching the text document for the corrected text mayinclude scoring a portion of the text document and comparing the scoreof the portion to an empirically determined threshold value to determinewhether the portion of the text document includes a word that is not inthe vocabulary. In this case, the method may further include outputtinga result if it is determined that the portion of the text documentincludes a word that is not in the vocabulary. Moreover, the step ofoutputting the result may include highlighting the portion of the textdocument or re-recognizing the portion of the text document.

In another general aspect a computer-implemented method for speechrecognition includes receiving dictated text, generating recognizedspeech based on the received dictated text, receiving an edited text ofthe recognized speech, and determining an acoustic model for the editedtext. The step of generating includes determining acoustic models forthe dictated text that best match acoustic data for the dictated text.The edited text indicates a replacement for a portion of the dictatedtext. The method also includes determining whether to adapt acousticmodels for the edited text based on the acoustic model for the editedtext and the acoustic model for the dictated text portion.

Implementations may include one or more of the following features. Themethod may also include calculating an acoustic model score based on acomparison between the acoustic model for the edited text and theacoustic data for the dictated text portion. In this case, the step ofdetermining whether to adapt acoustic models for the edited text may bebased on the calculated acoustic model score. The step of determiningwhether to adapt acoustic models may include calculating an originalacoustic model-score based on a comparison between the acoustic modelfor the dictated text portion and the acoustic data for the dictatedtext portion. The step of determining whether to adapt acoustic modelsmay include calculating a difference between the acoustic model scoreand the original acoustic model score. The step of determining whetherto adapt acoustic models may include determining whether the differenceis less than a predetermined value. The step of determining whether toadapt acoustic models may include adapting acoustic models for theedited text if the difference is less than a predetermined value. Thestep of determining whether to adapt acoustic models for the edited textmay include bypassing adapting acoustic models for the edited text ifthe difference is greater than or equal to a predetermined value.

The step of receiving the edited text of the recognized speech may occurduring a recognition session in which the recognized speech is generatedor after a recognition session in which the recognized speech isgenerated. The step of receiving the edited text of the recognizedspeech may include receiving a selection of the portion of the dictatedtext.

The step of determining an acoustic model for the edited text mayinclude searching for the edited text in a vocabulary or a backupdictionary used to generate the recognized speech. The step ofdetermining an acoustic model for the edited text may include selectingan acoustic model that best matches the edited text.

In another general aspect, a computer-implemented method of speechrecognition includes performing speech recognition on an utterance toproduce a recognition result for the utterance, receiving a selection ofthe recognition result, receiving a correction of the recognitionresult, and performing speech recognition on the correction using aconstraint grammar that permits spelling and pronunciation in parallel.The method includes identifying whether the correction comprises aspelling or a pronunciation using the constraint grammar.

Implementations may include one or more of the following features. Themethod may include generating a replacement result for the recognitionresult based on the correction.

The constraint grammar may include a spelling portion and a dictationvocabulary portion. In that case, the spelling portion may indicate thatthe first utterance from the user is a letter in an alphabet. Thevocabulary portion may indicate that the first utterance from the useris a word from the dictation vocabulary. The spelling portion mayindicate a frequency with which letters occur in a language model. Thedictation vocabulary portion may indicate a frequency with which wordsoccur in a language model. The method may also include introducing abiasing value between the spelling and the dictation vocabulary portionsof the constraint grammar.

Systems and computer programs for implementing the described techniquesand systems are also contemplated.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features,objects, and advantages will be apparent from the description, thedrawings, and the claims.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a speech recognition system.

FIG. 2 is a block diagram of speech recognition software of the systemof FIG. 1.

FIGS. 3A and 3B are state diagrams of a constraint grammar.

FIG. 4 is a flow chart of a speech recognition procedure.

FIG. 5 is a block diagram of a speech recognition system.

FIGS. 6-8 are block diagrams of other implementations of the system ofFIG. 5.

FIG. 9 is a block diagram of a recorder of the system of FIG. 5.

FIG. 10 is a block diagram of a computer of the system of FIG. 5.

FIGS. 11A-11C are screen displays of a user interface of the speechrecognition system of FIGS. 1 and 5.

FIGS. 12 and 16 are flow charts of procedures implemented by a speechrecognition system such as the system shown in FIG. 5.

FIG. 13 is a block diagram of a procedure for retrieving transcriptsfrom a speech recognition result determined using the procedures ofFIGS. 12 and 16.

FIGS. 14, 17, 18, and 19A-19C are screen displays of a user interface ofthe speech recognition system of FIGS. 1 and 5.

FIG. 15 is a table showing synchronization between first and alternativetranscripts used to determine a choice list using the procedures of FIG.12 and 16.

FIG. 20 is a flow chart of a procedure implemented by a speechrecognition system such as the system shown in FIG. 5.

FIG. 21 is a table showing a phoneme confusability matrix.

FIG. 22 is a diagram showing correction of a word using the procedure ofFIG. 20.

FIG. 23 is a flow chart of a procedure implemented by a speechrecognition system such as the system shown in FIG. 5.

FIGS. 24A and 24B are graphs showing correction of errors using theprocedure of FIG. 23 in comparison to random editing.

FIG. 25 is a flow chart of a procedure implemented by a speechrecognition system such as the system shown in FIG. 5.

FIG. 26 shows a constraint grammar used in the procedure of FIG. 25.

Like reference symbols in the various drawings indicate like elements.

DESCRIPTION

Referring to FIG. 1, one implementation of a speech recognition system100 includes input/output (I/O) devices (for example, microphone 105,mouse 110, keyboard 115, and display 120) and a general-purpose computer125 having a processor 130, an I/O unit 135 and a sound card 140. Amemory 145 stores data and programs such as an operating system 150, anapplication program 155 (for example, a word processing program), andspeech recognition software 160.

The microphone 105 receives the user's speech and conveys the speech, inthe form of an analog signal, to the sound card 140, which in turnpasses the signal through an analog-to-digital (A/D) converter totransform the analog signal into a set of digital samples. Under controlof the operating system 150 and the speech recognition software 160, theprocessor 130 identifies utterances in the user's continuous speech.Utterances are separated from one another by a pause having asufficiently large, predetermined duration (for example, 160-250milliseconds). Each utterance may include one or more words of theuser's speech.

The system may also include an analog recorder port 165 and/or a digitalrecorder port 170. The analog recorder port 165 is connected to thesound card 140 and is used to transmit speech recorded using a hand-heldrecorder to the sound card. The analog recorder port may be implementedas a microphone positioned to be next to the speaker of the hand-heldrecorder when the recorder is inserted into the port 165, and may beimplemented using the microphone 105. Alternatively, the analog recorderport 165 may be implemented as a tape player that receives a taperecorded using a hand-held recorder and transmits information recordedon the tape to the sound card 140.

The digital recorder port 170 may be implemented to transfer a digitalfile generated using a hand-held digital recorder. This file may betransferred directly into memory 145. The digital recorder port 170 maybe implemented as a storage device (for example, a floppy drive orCD-ROM drive) of the computer 125.

FIG. 2 illustrates typical components of the speech recognition software160. For ease of discussion, the following description indicates thatthe components carry out operations to achieve specified results.However, it should be understood that each component actually causes theprocessor 130 to operate in the specified manner.

Initially, a front end processing module 200 converts the digitalsamples 205 from the sound card 140 (or from the digital recorder port170) into frames of parameters 210 that represent the frequency contentof an utterance. Each frame includes 24 parameters and represents ashort portion (for example, 10 milliseconds) of the utterance.

A recognizer 215 receives and processes the frames of an utterance toidentify text corresponding to the utterance. The recognizer entertainsseveral hypotheses about the text and associates a score with eachhypothesis. The score reflects the probability that a hypothesiscorresponds to the user's speech. For ease of processing, scores aremaintained as negative logarithmic values. Accordingly, a lower scoreindicates a better match (a high probability) while a higher scoreindicates a less likely match (a lower probability), with the likelihoodof the match decreasing as the score increases. After processing theutterance, the recognizer provides the best-scoring hypotheses to thecontrol/interface module 220 as a list of recognition candidates, whereeach recognition candidate corresponds to a hypothesis and has anassociated score. Some recognition candidates may correspond to textwhile other recognition candidates correspond to commands. Commands mayinclude words, phrases, or sentences.

The recognizer 215 processes the frames 210 of an utterance in view ofone or more constraint grammars 225. A constraint grammar, also referredto as a template or restriction rule, may be a limitation on the wordsthat may correspond to an utterance, a limitation on the order orgrammatical form of the words, or both. For example, a constraintgrammar for menu-manipulation commands may include only entries from themenu (for example, “file”, “edit”) or command words for navigatingthrough the menu (for example, “up”, “down”, “top”, “bottom”). Differentconstraint grammars may be active at different times. For example, aconstraint grammar may be associated with a particular applicationprogram 155 and may be activated when the user opens the applicationprogram and deactivated when the user closes the application program.The recognizer 215 discards any hypothesis that does not comply with anactive constraint grammar. In addition, the recognizer 215 may adjustthe score of a hypothesis associated with a particular constraintgrammar based on characteristics of the constraint grammar.

FIG. 3A illustrates an example of a constraint grammar for a “select”command used to select previously recognized text. As shown, aconstraint grammar may be illustrated as a state diagram 400. The“select” command includes the word “select” followed by one or morepreviously-recognized words, with the words being in the order in whichthey were previously recognized. The first state 405 of the constraintgrammar indicates that the first word of the select command must be“select”. After the word “select”, the constraint grammar permits atransition along a path 410 to a second state 415 that requires the nextword in the command to be a previously-recognized word. A path 420,which returns to the second state 415, indicates that the command mayinclude additional previously-recognized words. A path 425, which exitsthe second state 415 and completes the command, indicates that thecommand may include only previously-recognized words. FIG. 3Billustrates the state diagram 450 of the constraint grammar for theselect command when a previously-recognized utterance is “four score andseven”. This state diagram could be expanded to include words fromadditional utterances. The “select” command and techniques forgenerating its constraint grammar are described further in U.S. Pat. No.5,794,189, entitled “CONTINUOUS SPEECH RECOGNITION” and issued Aug. 11,1998, which is incorporated herein by reference.

The constraint grammar also may be expressed in Backus-Naur Form (BNF)or Extended BNF (EBNF). In EBNF, the grammar for the “Select” commandis:

-   -   <recognition result>::=Select <words>,

where

-   -   <words>::=[PRW1[PRW2[PRW3 . . . PRWn]]]|    -   [PRW2[PRW3 . . . PRWn]] | . . . [PRWn],    -   “PRWi” is the previously-recognized word i,    -   [ ] means optional,    -   < > means a rule,    -   51 means an OR function, and    -   ::=means “is defined as” or “is”.

As illustrated in FIGS. 3A and 3B, this notation indicates that “select”may be followed by any ordered sequence of previously-recognized words.This grammar does not permit optional or alternate words. In someinstances, the grammar may be modified to permit optional words (forexample, an optional “and” to permit “four score and seven” or “fourscore seven”) or alternate words or phrases (for example, “four scoreand seven” or “eighty seven”). Constraint grammars are discussed furtherin U.S. Pat. No. 5,799,279, entitled “CONTINUOUS SPEECH RECOGNITION OFTEXT AND COMMANDS” and issued Aug. 25, 1998, which is incorporatedherein by reference.

Another constraint grammar 225 that may be used by the speechrecognition software 160 is a large vocabulary dictation grammar. Thelarge vocabulary dictation grammar identifies words included in theactive vocabulary 230, which is the vocabulary of words available to thesoftware during recognition. The large vocabulary dictation grammar alsoindicates the frequency with which words occur. A language modelassociated with the large vocabulary dictation grammar may be a unigrammodel that indicates the frequency with which a word occursindependently of context, or a bigram model that indicates the frequencywith which a word occurs in the context of a preceding word. Forexample, a bigram model may indicate that a noun or adjective is morelikely to follow the word “the” than is a verb or preposition.

Other constraint grammars 225 include an in-line dictation macrosgrammar for dictation commands, such as “CAP” or “Capitalize” tocapitalize a word and “New-Paragraph” to start a new paragraph; theselect X Y Z grammar discussed above and used in selecting text; anerror correction commands grammar; a dictation editing grammar; anapplication command and control grammar that may be used to control aparticular application program 155; a global command and control grammarthat may be used to control the operating system 150 and the speechrecognition software 160; a menu and dialog tracking grammar that may beused to manipulate menus; and a keyboard control grammar that permitsthe use of speech in place of input devices, such as the keyboard 115 orthe mouse 110.

The active vocabulary 230 uses a pronunciation model in which each wordis represented by a series of phonemes that comprise the phoneticspelling of the word. Each phoneme may be represented as a triphone thatincludes multiple nodes. A triphone is a context-dependent phoneme. Forexample, the triphone “abc” represents the phoneme “b” in the context ofthe phonemes “a” and “c”, with the phoneme “b” being preceded by thephoneme “a” and followed by the phoneme “c”.

One or more vocabulary files may be associated with each user. Thevocabulary files contain all of the words, pronunciations, and languagemodel information for the user. Dictation and command grammars may besplit between vocabulary files to optimize language model informationand memory use, and to keep each single vocabulary file under 64,000words.

Separate acoustic models 235 are provided for each user of the system.Initially speaker-independent acoustic models of male or female speechare adapted to a particular user's speech using an enrollment program.The acoustic models may be further adapted as the system is used. Theacoustic models are maintained in a file separate from the activevocabulary 230.

The acoustic models 235 represent phonemes. In the case of triphones,the acoustic models 235 represent each triphone node as a mixture ofGaussian probability density functions (“PDFs”). For example, node “i”of a triphone “abc” may be represented as ab^(i)c:${{{ab}^{i}c} = {\sum\limits_{k}{w_{k}{N\left( {\mu_{k},c_{k}} \right)}}}},$where each W_(k) is a mixture weight, ${{\sum\limits_{k}w_{k}} = 1},$μ_(k) is a mean vector for the probability density function (“PDF”)N_(k), and C_(k) is the covariance matrix for the PDF N_(k). Like theframes in the sequence of frames, the vectors μ_(k) each include twentyfour parameters. The matrices c_(k) are twenty four by twenty fourmatrices. Each triphone node may be represented as a mixture of up to,for example, sixteen different PDFs.

A particular PDF may be used in the representation of multiple triphonenodes. Accordingly, the acoustic models 235 represent each triphone nodeas a collection of mixture weights w_(k) associated with up to sixteendifferent PDFs N_(k) and separately represent each PDF N_(k) using amean vector μ_(k) and a covariance matrix c_(k). Use of a particular PDFto represent multiple triphone nodes permits the models to include asmaller number of PDFs than would be required if each triphone nodeincluded entirely separate PDFs. Since the English language may beroughly represented using 50 different phonemes, there may be up to125,000 (50³) different triphones, which would result in a huge numberof PDFs if each triphone node were represented by a separate set ofPDFs. Representing multiple nodes with common PDFs also may remedy orreduce a data sparsity problem that results because some triphones (forexample, “tzp” in the English language) rarely occur. These raretriphones may be represented by having closely-related triphones sharethe same set of PDFs.

A large vocabulary dictation grammar may include multiple dictationtopics (for example, “medical” or “legal”), each having its ownvocabulary file and its own language model. A dictation topic includes aset of words, which represents the active vocabulary 230, as well as anassociated language model.

A complete dictation vocabulary may consist of the active vocabulary 230plus a backup vocabulary 245. The backup vocabulary may include filesthat contain user-specific backup vocabulary words and system-widebackup vocabulary words.

User-specific backup vocabulary words include words that a user hascreated while using the speech recognition software. These words arestored in vocabulary files for the user and for the dictation topic, andare available as part of the backup dictionary for the dictation topicregardless of user, and to the user regardless of which dictation topicis being used. For example, if a user is using a medical topic and addsthe word “ganglion” to the dictation vocabulary, any other user of themedical topic will have immediate access to the word “ganglion”. Inaddition, the word will be written into the user-specific backupvocabulary. Then, if the user says “ganglion” while using a legal topic,the word “ganglion” will be available during correction from the backupdictionary.

In addition to the user-specific backup vocabulary noted above, there isa system-wide backup vocabulary. The system-wide backup vocabularycontains all the words known to the system, including words that maycurrently be in an active vocabulary.

The recognizer 215 may operate in parallel with a pre-filteringprocedure 240. Upon initiating processing of an utterance, therecognizer 215 requests from the pre-filtering procedure 240 a list ofwords that may have been spoken as the first word of the utterance (thatis, words that may correspond to the first and subsequent frames of theutterance). The pre-filtering procedure 240 performs a coarse comparisonof the sequence of frames with the active vocabulary 230 to identify asubset of the vocabulary for which a more extensive comparison using therecognizer is justified.

After the pre-filtering procedure responds with the requested list ofwords, the recognizer initiates a hypothesis for each word from the listand compares acoustic models for the word to the frames of parametersrepresenting the utterance. The recognizer uses the results of thesecomparisons to generate scores for the hypotheses. Hypotheses havingexcessive scores are eliminated from further consideration. As notedabove, hypotheses that comply with no active constraint grammar also areeliminated.

When the recognizer determines that a word of a hypothesis has ended,the recognizer requests from the pre-filtering procedure a list of wordsthat may have been spoken just after the ending-time of the word. Therecognizer then generates a new hypothesis for each word on the list,where each new hypothesis includes the words of the old hypothesis plusthe corresponding new word from the list.

In generating the score for a hypothesis, the recognizer uses acousticscores for words of the hypothesis, a language model score thatindicates the likelihood that words of the hypothesis are used together,and scores provided for each word of the hypothesis by the pre-filteringprocedure. The recognizer may eliminate any hypothesis that isassociated with a constraint grammar (for example, a commandhypothesis), but does not comply with the constraint grammar.

Referring to FIG. 4, the recognizer 215 may operate according to aprocedure 1200. First, prior to processing, the recognizer 215initializes a lexical tree (step 1205). The recognizer 215 thenretrieves a frame of parameters (step 1210) and determines whether thereare hypotheses to be considered for the frame (step 1215). The firstframe always corresponds to silence so that there are no hypotheses tobe considered for the first frame.

If hypotheses need to be considered for the frame (step 1215), therecognizer 215 goes to the first hypothesis (step 1220). The recognizerthen compares the frame to acoustic models 235 for the last word of thehypothesis (step 1225) and, based on the comparison, updates a scoreassociated with the hypothesis (step 1230).

After updating the score (step 1230), the recognizer determines whetherthe user was likely to have spoken the word or words corresponding tothe hypothesis (step 1235). The recognizer makes this determination bycomparing the current score for the hypothesis to a threshold value. Ifthe score exceeds the threshold value, then the recognizer 215determines that the hypothesis is too unlikely to merit furtherconsideration and deletes the hypothesis (step 1240).

If the recognizer determines that the word or words corresponding to thehypothesis were likely to have been spoken by the user, then therecognizer determines whether the last word of the hypothesis is ending(step 1245). The recognizer determines that a word is ending when theframe corresponds to the last component of the model for the word. Ifthe recognizer determines that a word is ending (step 1245), therecognizer sets a flag that indicates that the next frame may correspondto the beginning of a word (step 1250).

If there are additional hypotheses to be considered for the frame (step1255), then the recognizer selects the next hypothesis (step 1260) andrepeats the comparison (step 1225) and other steps. If there are no morehypotheses to be considered for the frame (step 1255), then therecognizer determines whether there are more frames to be considered forthe utterance (step 1265). The recognizer determines that there are moreframes to be considered when two conditions are met. First, more framesmust be available. Second, the best scoring node for the current frameor for one or more of a predetermined number of immediately precedingframes must have been a node other than the silence node (that is, theutterance has ended when the silence node is the best scoring node forthe current frame and for a predetermined number of consecutivepreceding frames).

If there are more frames to be considered (step 1265) and the flagindicating that a word has ended is set (step 1270), or if there were nohypotheses to be considered for the frame (step 1215), then therecognizer requests from the pre-filtering procedure 240 a list of wordsthat may start with the next frame (step 1275). Upon receiving the listof words from the pre-filtering procedure, the recognizer uses the listof words to create hypotheses or to expand any hypothesis for which aword has ended (step 1280). Each word in the list of words has anassociated score. The recognizer uses the list score to adjust the scorefor the hypothesis and compares the result to a threshold value. If theresult is less than the threshold value, then the recognizer maintainsthe hypothesis. Otherwise, the recognizer determines that the hypothesisdoes not merit further consideration and abandons the hypothesis. As anadditional part of creating or expanding the hypotheses, the recognizercompares the hypotheses to the active constraint grammars 225 andabandons any hypothesis that corresponds to no active constraintgrammar. The recognizer then retrieves the next frame (step 1210) andrepeats the procedure.

If there are no more speech frames to process, then the recognizer 215provides the most likely hypotheses to the control/interface module 220as recognition candidates (step 1285).

The control/interface module 220 controls operation of the speechrecognition software and provides an interface to other software or tothe user. The control/interface module receives the list of recognitioncandidates for each utterance from the recognizer. Recognitioncandidates may correspond to dictated text, speech recognition commands,or external commands. When the best-scoring recognition candidatecorresponds to dictated text, the control/interface module provides thetext to an active application, such as a word processor. Thecontrol/interface module also may display the best-scoring recognitioncandidate to the user through a graphical user interface. When thebest-scoring recognition candidate is a command, the control/interfacemodule 220 implements the command. For example, the control/interfacemodule may control operation of the speech recognition software inresponse to speech recognition commands (for example, “wake up”, “makethat”), and may forward external commands to the appropriate software.

The control/interface module also controls the active vocabulary,acoustic models, and constraint grammars that are used by therecognizer. For example, when the speech recognition software is beingused in conjunction with a particular application (for example,Microsoft Word), the control/interface module updates the activevocabulary to include command words associated with that application andactivates constraint grammars associated with the application.

Other functions provided by the control/interface module 220 may includea vocabulary customizer and a vocabulary manager. The vocabularycustomizer optimizes the language model of a specific topic by scanninguser supplied text. The vocabulary manager is a developer tool that isused to browse and manipulate vocabularies, grammars, and macros. Eachsuch function of the control/interface module 220 may be implemented asan executable program that is separate from the main speech recognitionsoftware. Similarly, the control/interface module 220 also may beimplemented as a separate executable program.

The control/interface module 220 also may provide an enrollment programthat uses an enrollment text and a corresponding enrollment grammar tocustomize the speech recognition software to a specific user. Theenrollment program may operate in an interactive mode that guides theuser through the enrollment process, or in a non-interactive mode thatpermits the user to enroll independently of the computer. In theinteractive mode, the enrollment program displays the enrollment text tothe user and the user reads the displayed text. As the user reads, therecognizer 215 uses the enrollment grammar to match a sequence ofutterances by the user to sequential portions of the enrollment text.When the recognizer 215 is unsuccessful, the enrollment program promptsthe user to repeat certain passages of the text. The recognizer usesacoustic information from the user's utterances to train or adaptacoustic models 235 based on the matched portions of the enrollmenttext. One type of interactive enrollment program is discussed in U.S.Pat. No. 6,212,498, entitled “ENROLLMENT IN SPEECH RECOGNITION” andissued Apr. 3, 2001, which is incorporated herein by reference.

In the non-interactive mode, the user reads the text without promptingfrom the computer. This offers the considerable advantage that, inaddition to reading text displayed by the computer, the user can readfrom a printed text independent of the computer. Thus, the user couldread the enrollment text into a portable recording device and laterdownload the recorded information into the computer for processing bythe recognizer. In addition, the user is not required to read every wordof the enrollment text, and may skip words or paragraphs as desired. Theuser also may repeat portions of the text. This adds substantialflexibility to the enrollment process.

The enrollment program may provide a list of enrollment texts, each ofwhich has a corresponding enrollment grammar, for the user's selection.Alternatively, the user may input an enrollment text from anothersource. In this case, the enrollment program may generate the enrollmentgrammar from the input enrollment text, or may employ a previouslygenerated enrollment grammar.

The control/interface module 220 may also implement error correction andcursor/position manipulation procedures of the software 160. Errorcorrection procedures include a “make that” command and a “spell that”command. Cursor/position manipulation procedures include the “select”command discussed above and variations thereof (for example, “select[start] through [end]”), “insert before/after” commands, and a “resumewith” command.

During error correction, word searches of the backup vocabularies startwith the user-specific backup dictionary and then check the system-widebackup dictionary. The backup dictionaries also are searched when thereare new words in text that a user has typed.

When the system makes a recognition error, the user may invoke anappropriate correction command to remedy the error. Various correctioncommands are discussed in U.S. Pat. No. 5,794,189, entitled “CONTINUOUSSPEECH RECOGNITION” and issued Aug. 11, 1998, U.S. Pat. No. 6,064,959,entitled “ERROR CORRECTION IN SPEECH RECOGNITION” and issued May 16,2000, and U.S. application Ser. No. 09/094,611, entitled “POSITIONMANIPULATION IN SPEECH RECOGNITION” and filed Jun. 15, 1998, all ofwhich are incorporated herein by reference.

Referring to FIG. 5, the speech recognition system may be implementedusing a system 1400 for performing recorded actions that includes apocket-sized recorder 1405 and a computer 1410 (not shown to scale).When data is to be transmitted, the recorder 1405 may be connected tothe computer 1410 using a cable 1415. Other data transmissiontechniques, such as infrared data transmission, also may be used.

In the described implementation, the recorder 1405 is a digital recorderhaving time stamp capabilities. One recorder meeting these criteria isthe Dragon Naturally Mobile Pocket Recorder RI manufactured for DragonSystems, Inc., of Newton, Mass. by Voice It Worldwide, Inc. In otherimplementations, the recorder may be a digital recorder lacking timestamp capabilities, or an analog recorder using a magnetic tape.

FIG. 6 illustrates a variation 1400A of the system in which an outputdevice 1420 is attached to the recorder 1405. Information about actionitems recorded using the recorder 1405 and processed by the computer1410 is transferred automatically via the cable 1415 for display on theoutput device 1420. This variation permits the user to access, forexample, appointments and contact information using the display 1420.Keys 1425 on the recorder are used to navigate through displayedinformation.

FIG. 7 illustrates another variation 1400B in which the recording andoutput functionality are implemented using a PDA or a hand-held computer1430. With this variation, it is contemplated that some instances of thehand-held computer 1430 may have sufficient processing capacity toperform some or all of the speech recognition, parsing, and otherprocessing tasks described below.

FIG. 8 illustrates another variation 1400C in which the user's speech isimmediately transmitted to the computer 1410 using, for example, acellular telephone 1435. This variation permits the user to dictateactions over an extended period that might exceed the capacity of arecorder. Audio feedback may be provided to permit immediate review ofan action item, interactive correction, and performance of the actionitem. The interactive correction may be provided using spoken commands,telephone key strokes, or a combination of the two.

Referring also to FIG. 9, the recorder 1405 includes a record button1500 that activates the recorder, a microphone 1505 that converts auser's speech into an analog electrical signal, an analog-to-digitalconverter 1510 that converts the analog electrical signal into a seriesof digital samples, a processor 1515, a memory 1520, and an output port1525 for connection to the cable 1415. When the user presses the recordbutton 1500 and speaks into the microphone 1505, the processor creates afile 1530 in memory 1520 and stores in the file a time stamp 1535corresponding to the time at which the button was pressed in the file.The processor then stores the digital samples 1540 corresponding to theuser's speech in the same file. In some implementations, the processoruses compression techniques to compress the digital samples to reducestorage and data transfer requirements. The user may use the recordermultiple times before transferring data to the computer 1410.

Referring also to FIG. 10, the computer 1410 may be a standard desktopcomputer. In general, such a computer includes input/output (I/O)devices (for example, microphone 1605, mouse 1610, keyboard 1615, anddisplay 1620) and a console 1625 having a processor 1630, an I/O unit1635 and a sound card 1640. A memory 1645 stores data and programs suchas an operating system 1650, an application program 1655 (for example, aword processing program), and speech recognition software 1660.

The computer 1410 may be used for traditional speech recognition. Inthis case, the microphone 1605 receives the user's speech and conveysthe speech, in the form of an analog signal, to the sound card 1640,which in turn passes the signal through an analog-to-digital (A/D)converter to transform the analog signal into a set of digital samples.Under control of the operating system 1650 and the speech recognitionsoftware 1660, the processor 1630 identifies utterances in the user'scontinuous speech. Utterances are separated from one another by a pausehaving a sufficiently large, predetermined duration (for example,160-250 milliseconds). Each utterance may include one or more words ofthe user's speech.

The system also includes a digital recorder port 1665 and/or an analogrecorder port 1670 for connection to the cable 1415. The digitalrecorder port 1665 is used to transfer files generated using therecorder 1405. These files may be transferred directly into memory 1645,or to a storage device such as hard drive 1675. The analog recorder port1670 is connected to the sound card 1640 and is used to transmit speechrecorded using an analog or digital recorder to the sound card. Theanalog recorder port may be implemented using a line in port. Thehand-held recorder is connected to the port using a cable connectedbetween the line in port and a line out or speaker port of the recorder.The analog recorder port also may be implemented using a microphone,such as the microphone 1605. Alternatively, the analog recorder port1670 may be implemented as a tape player that receives a tape recordedusing a hand-held recorder and transmits information recorded on thetape to the sound card 1640.

To implement the speech recognition and processing functions of thesystem 1400, the computer 1410 runs interface software 1680, the speechrecognition software 1660, a parser 1685, and back-end software 1690.Dragon NaturallySpeaking Preferred Edition 3.1, available from DragonSystems, Inc. of Newton, Mass., offers one example of suitable speechrecognition software. The interface software 1680 provides a userinterface for controlling the transfer of data from the digital recorderand the generation of action items for use by the back-end software1690. In general, the user interface may be controlled using inputdevices such as a mouse or keyboard, or using voice commands processedby the speech recognition software.

After transferring data from the recorder, the interface software 1680provides the digital samples for an action item to the speechrecognition software 1660. If the digital samples have been stored usingcompression techniques, the interface software 1680 decompresses themprior to providing them to the speech recognition software. In general,the speech recognition software analyzes the digital samples to producea sequence of text, and provides this sequence to the interface software1680. The interface software 1680 then transfers the text and theassociated time stamp, if any, to the parser 1685, which processes thetext in conjunction with the time stamp to generate a parsed version ofthe action item. The parser returns the parsed action item to theinterface software, which displays it to the user. After any editing bythe user, and with user approval, the interface software then transfersthe action item to the appropriate back-end software 1690. An example ofback-end software with which the system works is personal informationmanagement software, such as Microsoft Outlook, which is available fromMicrosoft Corporation of Redmond, Wash. Other suitable back-end softwareincludes contact management software, time management software, expensereporting applications, electronic mail programs, and fax programs.

Various systems for recognizing recorded speech and performing actionsidentified in the speech are discussed in U.S. Application No.09/432,155, entitled “PERFORMING RECORDED ACTIONS” and filed Jun. 10,1999, which is incorporated herein by reference.

A user may dictate a document into an audio recorder such as recorder1405 and then may download the dictated audio information into a speechrecognition system like the one described above. Likewise, the user maydictate a document directly into a microphone connected to the speechrecognition system, which may be implemented in a desktop computer or ahand-held electronic device.

In a large vocabulary continuous speech recognition system, the user maycorrect misrecognition errors by selecting a range of characters fromthe speech recognition results. The speech recognition system presents alist of alternative recognitions for that selected range of charactersby, for example, opening the correction window with a choice list.

This type of error correction is used in Dragon NaturallySpeaking™ andother commercial large vocabulary continuous speech recognition systemscurrently on the market. Correction in speech recognition systemstypically requires the user to perform two steps. First, the useridentifies the range of words that are incorrect, which may be referredto as an error-filled region. The error-filled region includes abeginning character position and an ending character position. Second,the user selects a replacement from a list of alternatives for theselected error-filled region.

Correction in the speech recognition system may include a feature called“double click to correct,” in which the user double clicks on the firstword of the error-filled region in order to correct two or more words inthe recognition result. (In a system, such as one employing a handhelddevice, in which a stylus is used instead of a mouse, this feature maybe implemented by tapping, or double tapping, the stylus on the firstword in the error-filled region.) The speech recognition systemautomatically selects n words from the user's document beginning withthe word that was selected, where n is a predetermined integer thatindicates the number of selected words. In an implementation in which nequals three, the speech recognition system displays a list ofalternative recognition results, where each alternative recognitionresult replaces the three words that begin at the location of the wordthe user selected.

Although the double-click-to-correct feature relieves the user of theburden of having to select the end of the error-filled region, the endof the error-filled region is always computed to be the end of the groupof n words including the word that was selected. Accordingly, theselected range of words to be corrected (that is, n words including theselected word) may be larger than the actual error-filled region, thuscomplicating the error correction process. In some cases, the selectedrange of words to be corrected (n words including the selected word) maybe smaller than the actual error-filled region, thus forcing the user tocancel the list of alternatives and directly reselect the appropriaterange of characters.

The following description provides a discussion of additional systemsand methods that may be implemented to further improve error correctionin speech recognition. These additional systems and methods may beimplemented to correct errors in any speech recognition environment, andare not limited to the speech recognition systems described in detailand referenced above.

Choice List for Recognition Results

In the example shown in FIG. 11 A, the recognizer 215 misrecognizes thesentence “let's recognize speech” and the control/interface module 220responds by inserting the incorrect text “let's wreck a nice beach” 1700in dictation window 1702. In a conventional speech recognition system,as shown in FIG. 11B, the user causes the control/interface module 220to generate a choice list 1705 by selecting the word “wreck” 1710 in therecognition result 1700. The choice list 1705 includes a list ofalternative recognition candidates for the word “wreck” 1710.

A speech recognition system may determine the error-filled region on thefly during correction. In this way, the user selects (by clicking,double-clicking, tapping, double tapping, or in some other way) thefirst word in an error-filled region and the speech recognition systemautomatically computes a width of the error-filled region to determinealternative recognition results. The number of words in each of thealternative recognition results in the choice list varies (that is, thelength of each of the elements in the choice list is floating) becausethere is no rigidly defined end to the error-filled region.

In FIG. 11C, a speech recognition system has provided an improved list1720 (also referred to as a “floating choice list”) of alternatives forthe selected word (“wreck”) 1710. The improved list 1720 includesalternatives for the selected word 1710 along with alternatives for oneor more words following “wreck” in the document. In this way, the userneed not identify the end of an error-filled region. For example, thefirst entry 1730 in the choice list is “recognize speech.”

Referring to FIG. 12, the speech recognition system performs a procedure1800 for providing the floating choice list. Initially, the speechrecognition system receives input from a user indicating an incorrectword in an original transcript (step 1805). For example, the user mayposition a cursor on the screen over a word to select the incorrectword. The speech recognition system converts the screen coordinate intoa character position in the original transcript. Then, using thatcharacter position, the speech recognition system finds the beginning ofthe word that includes that character position—this word corresponds tothe incorrect word.

The speech recognition system retrieves a list of transcripts based onthe indicated incorrect word (step 1810). The speech recognition systemaccomplishes this retrieval by first retrieving a result object thatcreated the incorrect word and includes the character position. Eachtranscript includes a sequence of words and start times (called indextimes), where a start time is associated with each word in thetranscript. The index time may be given in units of millisecondsrelative to the start of an utterance.

For example, referring to FIG. 13, a result object 1900 is retrievedfrom an array 1905 of result objects for an original transcript 1910,where each result object 1900 describes a recognition. A list 1915 oftranscripts for result object 1900 is retrieved. Each transcript in thelist includes a set of one or more words (W_(ij)) and associated indextimes (t_(ij)), where the index i indicates the transcript and the indexj indicates the word in the transcript. The first (or original)transcript in the list 1915 of transcripts corresponds to thebest-scoring recognition result presented to the user. The remainingtranscripts in the list 1915 correspond to alternative transcripts thatwill be compared to the original transcript in subsequent analysis bythe speech recognition system.

Referring again to FIG. 12, after the list of transcripts is retrieved(step 1810), the speech recognition system analyzes the originaltranscript to determine the index time of the incorrect word (step1815). The speech recognition system then selects one of the alternativetranscripts from the list of transcripts for analysis (step 1820). Inone implementation, the speech recognition system selects the nextbest-scoring alternative transcript from the list of transcripts.

After the alternative transcript is selected (step 1820), the speechrecognition system analyzes the alternative transcript (step 1825) bysearching for an end of an error-filled region that begins with a wordwhose index time most closely matches that of the incorrect wordselected by the user. As discussed in detail below, the speechrecognition system searches for the location at which that alternativeresult transcript resynchronizes, or matches in time, with the originaltranscript. The speech recognition system searches forward in both theoriginal transcript and the alternative transcript until the systemfinds a word that is the same in both transcripts and that begins atapproximately the same time in both transcripts. If the speechrecognition system finds such a word, then the speech recognition systemproduces a replacement result that extends from the selected word to thematching word. The speech recognition system may also produce areplacement result when the incorrect word is positioned near the end ofthe original transcript, with the replacement result extending from theselected word to the end of the transcript.

If the speech recognition system produces a replacement result (step1830), the speech recognition system compares the replacement result toother replacement results (step 1840). If the replacement result has notbeen encountered before (step 1840), the speech recognition system savesthe replacement result to the choice list (step 1845) and checks foradditional alternative transcripts (step 1850). The system also checksfor additional alternative transcripts (step 1850) if the replacementresult has been encountered before and, therefore, is not saved (step1840), or if the speech recognition system does not produce areplacement result (step 1830).

If there are additional alternative transcripts (step 1850), the speechrecognition system selects a next alternative transcript (step 1855) foranalysis (step 1825). If there are no additional alternative transcriptsfor analysis (step 1850), the speech recognition system presents thechoice list (step 1860) and performs post-presentation updating (step1865).

Referring to FIGS. 14 and 15, for example, the speech recognition systemhas recognized the user's utterance as “I am dictating about the newYorkshire taste which is delicious,” as indicated in the dictationwindow 2000. The user has selected the word “new” 2005 in dictationwindow 2000, thus indicating that “new” is a word to be corrected. Thespeech recognition system has retrieved an original transcript “I amdictating about the new Yorkshire taste which is delicious” 2100 and analternative transcript “I am dictating about the New York shirt tastewhich is delicious” 2105.

In FIG. 15, the index times 2110 of the words of the original transcriptare shown below the words of the original transcript 2100 and the indextimes 2115 of the words in the alternative transcript 2105 are shownbelow the words of the alternative transcript. After the occurrence ofthe word “new,” the alternative transcript resynchronizes with theoriginal transcript at the word “taste” because the word “taste” intranscript 2100 and the word “taste” in transcript 2105 occur atapproximately the same index time. Thus, because the alternativetranscript resynchronizes with the original transcript at the word“taste,” the speech recognition system computes the end of theerror-filled region of the alternative transcript 2105 to be at the word“taste.”

As shown in FIG. 14, the speech recognition system produces a list ofreplacement results including replacement result “New York shirt” 2007for the transcript 2105 and presents the list of replacement results ina choice list 2010.

Referring also to FIG. 16, the speech recognition system performs aprocedure 1825 for analyzing the alternative transcript. First, thespeech recognition system finds a test word in the alternativetranscript that has an index time nearest to the index time of the wordto be corrected from the original transcript (step 2200). If the testword is identical to the word to be corrected (step 2205), then thespeech recognition system ignores the alternative transcript and exitsthe procedure 1825.

If the test word is not identical to the word to be corrected (step2205), then the speech recognition system designates a word immediatelyfollowing the word to be corrected in the original transcript as anoriginal transcript word and designates a word immediately following thetest word in the alternative transcript as an alternative transcriptword for subsequent analysis (step 2207). The speech recognition systemthen determines if the original transcript word is identical to thealternative transcript word (step 2210).

If the original transcript word is identical to the alternativetranscript word (step 2210), the speech recognition system computeswhether the index time of the original transcript word is near the indextime of the alternative transcript word (step 2215). If the index timesof the original transcript word and the alternative transcript word arenear each other (step 2215), then the speech recognition system extractsa replacement result that begins with the test word and ends with theword prior to the alternative transcript word (step 2220).

The required level of nearness between the index times may be controlledusing a parameter that may be manually adjusted and fine-tuned by adeveloper of the speech recognition system. For example, the system maycalculate a difference between index times for different words, and maydesignate index times as near each other when this difference is lessthan a threshold amount.

If the original transcript word is not identical to the alternativetranscript word (step 2210), then the speech recognition system computeswhether the index time of the original transcript word is near the indextime of the alternative transcript word (step 2225). If the index timesof the original transcript word and the alternative transcript word arenear each other (step 2225), then the speech recognition system selectsthe word adjacent to the original transcript word in the originaltranscript as the original transcript word for subsequent analysis andselects the word adjacent to the alternative transcript word in thealternative transcript as the alternative transcript word for subsequentanalysis (step 2230).

If the index time of the original transcript word is not near the indextime of the alternative transcript word (steps 2215 or 2225), the speechrecognition system computes whether the index time of the originaltranscript word is later than the index time of the alternativetranscript word (step 2235).

If the index time of the original transcript word is later than theindex time of the alternative transcript word (step 2235), then thespeech recognition system designates the word adjacent to thealternative transcript word in the alternative transcript as thealternative transcript word for subsequent analysis (step 2240). If theindex time of the original transcript word is not near (steps 2215 or2225) or is not later than (step 2235) the index time of the alternativetranscript word, then the index time of the original transcript word isearlier than the index time of the alternative transcript word (step2245). In this case, the speech recognition system selects the wordadjacent to the original transcript word in the original transcript asthe original transcript word for subsequent analysis (step 2250).

The example of FIGS. 14 and 15 will now be analyzed with respect to theprocedures 1800 and 1825. In FIG. 14, the speech recognition system hasreceived input from a user indicating that the word “new” 2005 from theoriginal transcript 2100 is to be corrected (step 1805). After selectingalternative transcript 2105 for examination (step 1820), the speechrecognition system finds the test word “New York” 2120 (where “New York”is a single lexical entry that is treated as a word by the system) inthe alternative transcript 2105 (step 2200). The test word “New York”has an index time of 1892, which is nearest the index time 1892 of theword “new.” Next, the speech recognition system compares the test word“New York” to the word “new” to determine that these words are notidentical (step 2205). Therefore, the speech recognition system sets theword “Yorkshire” as the original transcript word and sets the word“shirt” as the alternative transcript word (step 2207).

When the speech recognition system compares the word “Yorkshire” to theword “shirt” the speech recognition system determines that these wordsare not identical (step 2210). Furthermore, because the index time ofthe word “Yorkshire” is earlier than the index time of the word “shirt”(step 2245), the speech recognition system selects the word “taste,”which follows the word “Yorkshire” in the original transcript 2100, andhas an index time of 2729, as the original transcript word (step 2250).

At this point, the original transcript word is “taste” with an index of2729 and the alternative transcript word is “shirt” with an index of2490. Because the original transcript word and the alternativetranscript word are not identical (step 2210), and because the indextime of the original transcript word “taste” is later than the indextime of the alternative transcript word “shirt” (step 2235), the speechrecognition system selects the word “taste,” which has an index of 2809and follows “shirt” in the alternative transcript 2105, as thealternative transcript word (step 2240). At this point, the originaltranscript word is “taste” with an index of 2729 (from the originaltranscript 2100) and the alternative transcript word is “taste” with anindex of 2809 (from the alternative transcript 2105).

Because the original transcript word and the alternative transcript wordare identical to each other (step 2210), and because the index times ofthe original transcript word and the alternative transcript word arenear each other (step 2215), the speech recognition system extracts areplacement result from the alternative transcript 2105 that correspondsto “New York shirt” 2007 (step 2220).

Referring also to FIG. 17, dictation window 2300 is shown in which theuser has selected the word “the” to be corrected in the originaltranscript “I am dictating about the new Yorkshire taste which isdelicious.” In this case, the speech recognition system has provided achoice list 2305 that includes the replacement result “thee” and theoriginal result “the.”

In FIG. 18, for example, a dictation window 2400 is shown in which theuser has selected the word “taste” to be corrected in the originaltranscript “I am dictating about the new Yorkshire taste which isdelicious.” In this case, the speech recognition system has provided achoice list 2405 that includes the replacement results “paste,” “facedwitch's,” and “case to,” and the original result “taste.”

Upon receiving input from the user that indicates the word to becorrected, the speech recognition system may highlight the word to becorrected from the original transcript. For example, in FIG. 14, theword “new” is highlighted, in FIG. 17, the word “the” is highlighted,and in FIG. 18, the word “taste” is highlighted.

Referring again to FIG. 12, the speech recognition system performspost-presentation updating at step 1865 when the choice list ispresented to the user (step 1860). Post-presentation updating includesupdating the dictation window transcript to reflect a user selection ofa replacement result from the choice list. For example, referring alsoto FIGS. 19A-19C, the transcript may be updated with the replacementresult and the replacement result in the updated transcript may behighlighted when the user selects a replacement result from the choicelist. In FIG. 19A, the user selects “New York sure” 2500 from choicelist 2505 and the replacement result “New York sure” is highlighted inupdated transcript 2510. As shown in FIG. 19B, the user selects “NewYork shirt paste” 2515 from choice list 2505 and the replacement result“New York shirt paste” is highlighted in updated transcript 2520. InFIG. 19C, the user selects “New York shirt faced witch's” 2525 fromchoice list 2505 and the replacement result “New York shirt facedwitch's” 2525 is highlighted in updated transcript 2530.

As shown in FIGS. 17 and 18, replacement results produced by the speechrecognition system during procedure 1800 and shown in the choice listsmay include just a single word that either matches the originaltranscript or does not match the original transcript. Thus, the word“the,” which matches the original transcript in FIG. 17, and the word“thee,” which does not match the original transcript, are displayed inchoice list 2305.

Post-presentation updating (step 1865) may include ending a correctionsession upon receipt of an indication from the user that an alternativeresult reflects the user's original intent. For example, the speechrecognition system may terminate the correction session when the userclicks (or double clicks) a button that closes the choice list or whenthe user selects an appropriate alternative result.

Finding Multiple Misrecognitions of Utterances in a Transcript

A user may dictate a document into an audio recorder such as recorder1405 and then may download the dictated audio information into a speechrecognition system like the one described above. Likewise, the user maydictate a document directly into a microphone connected to the speechrecognition system, which may be implemented in a desktop computer or ahand-held electronic device. In either case, the speech recognitionsystem may be unable to recognize particular words that are not in thespeech recognition system's vocabulary. These words are referred to asout-of-vocabulary (OOV) words.

For example, the speech recognition system's vocabulary may not containproper names, such as the name “Fooberman,” or newer technical terms,such as the terms “edutainment” and “winsock.” When it encounters an OOVword, the speech recognition system may represent the word usingcombinations of words and phonemes in its vocabulary that most closelyresemble the OOV word. For example, the speech recognition system mayrecognize the word “Fooberman” as “glue bar man,” in which case thespeech recognition system has replaced the phoneme for “f” with thephonemes for “gl” and the phoneme “ûr” with the phoneme “âr”.

A user may proofread a text document representing recognized speech tocorrect OOV words and other misrecognitions within the text document.The user uses a keyboard, a mouse or speech to select what appears to bea misrecognized word, plays back the audio signal that produced theapparently misrecognized word, and then manually corrects themisrecognized word using, for example, a keyboard or speech. The userperforms this manual correction for each apparently misrecognized wordin the text document. The user must remain alert while reading over thetext document because it is sometimes difficult to detect amisrecognized word. This may be particularly important when detectingOOV words, which tend to be uncommon.

Optionally, the user (or the speech recognition system) may add the OOVword to the speech recognition system's vocabulary once the userrealizes that the system has misrecognized the word. The speechrecognition system may then re-recognize the whole text document using anew vocabulary that now includes what was previously an OOV word. Thisre-recognition process may take a relatively long time.

Referring also to FIG. 20, the speech recognition system maysubstantially reduce delays associated with correcting the OOV word byimplementing an OOV global correction according to a procedure 2600.Initially, the speech recognition system receives a general phonemeconfusability matrix (step 2605) and the text document representingrecognized speech (step 2610). The text document includes associatedlists of recognition candidates for each recognized utterance. The listsare created by the system during recognition.

The general phoneme confusability matrix is built before the procedureof FIG. 20 is implemented using the premise that any phoneme may beconfused for another phoneme. The probability of confusion depends onthe characteristics of the two phonemes and the characteristics of thespeaker's pronunciation. For example, the phoneme for “m” is commonlymistaken for the phoneme for “n”, and the phoneme for “t” is commonlymistaken for the phoneme for “d”.

FIG. 21 shows a general phoneme confusability matrix 2700 for a subsetof the phonemes in a one type of phonetic alphabet. Using the phoneticalphabet, for example, the phrase “the term of this agreement shallbegin on the 31st day of January, \comma 1994, \comma” may translateinto phonemes:

“D/tVm@vDis@grEm˜tS@lb/ginonD/TVt/fVstA@vjanyUer/kom@nIntEnnIn/f{rkom@”.

In the general phoneme confusability matrix 2700, scores for confusedpronunciation matches are represented as negative logarithms of a rateor likelihood that a spoken phoneme corresponding to the row isrecognized as the phoneme corresponding to the column. Therefore, ahigher number indicates a lower probability of confusion and a lowernumber indicates a higher probability of confusion. The phonemeconfusability matrix may be adapted continually for a particular user'sspeech patterns.

For example, the phoneme “z” is recognized as the phoneme “s” at a rateof e⁻¹⁵ or about 3×10⁻⁷, whereas the phoneme “i” is recognized as thephoneme “6” at a rate of e⁻¹⁰ or about 5×10⁻⁵. As another example, thephoneme “q” is confused with (or recognized correctly as) the phoneme“q” at a rate of e⁻¹ or about 0.4. This occurs because the speechrecognition system is often unable to identify the phoneme “q” inspeech.

The phoneme “<b>” listed in the matrix 2700 corresponds to a blank.Therefore, the entry for—w, <b>—represents the probability that thephoneme “w” is deleted, whereas the entry for—<b>, w—represents theprobability that the phoneme “w” is inserted. Thus, for example, thephoneme “t” is deleted at a rate of e⁻⁷ or 9×10⁻⁴ and the phoneme “e” ininserted at a rate of e⁻¹⁹ or 6×10⁻⁹.

A determination of which phonemes may be confused with each other andthe probability of that confusion may be based on empirical data. Suchempirical data may be produced, for example, by gathering a speechrecognition system's rate of confusion of phonemes for a preselectedpopulation or by studying frequency characteristics of differentphonemes. A speech recognition system also may gather this data forpronunciations by a single user as part of the system's continuoustraining.

Scores for confused pronunciation matches in the general phonemeconfusability matrix may be generated using three sources ofinformation: the probability that a sequence of phonemes for which thematches were sought (a recognized sequence) was the actual sequence ofphonemes produced by the speaker, the probability that a particularconfused pronunciation (the confused sequence) was confused for therecognized sequence, and the probability that the confused sequenceoccurs in the language (for example, English) with which the speechrecognition system is used. These probabilities correspond to the scoresproduced by, respectively, the recognizer for the recognized sequence, adynamic programming match of the recognized phonemes with the dictionarypronunciation using a priori probabilities of phoneme confusion, and anexamination of a unigram language model for the words corresponding tothe pronunciation of the recognized sequence.

Referring again to FIG. 20, the user is able to view the text documentusing a word processing program, or another program that displays thetext document. The user corrects mistakes found in the text document by,for example, typing or dictating the correct spelling of a word. In thisway, the user provides the speech recognition system with corrected text5 for a misrecognized word (step 2615). The speech recognition systemsearches the vocabulary for the corrected text (step 2620). If thecorrected text is in the vocabulary, the speech recognition systemawaits another correction from the user (step 2615).

If the corrected text is not in the vocabulary, the corrected text is anOOV word. In this case, the speech recognition system generates asequence of phonemes for the corrected text (step 2625). In generatingthe sequence of phonemes for the corrected text, the speech recognitionsystem uses a phonetic alphabet.

The speech recognition system aligns the phonemes for the corrected textwith the phonemes in each of the misrecognized words in the choice listfor the utterance that includes the corrected text (step 2630) and thenadjusts a copy of a general phoneme confusability matrix based on thealignment (step 2635). The speech recognition system searches therecognized speech for the OOV word using the adjusted phonemeconfusability matrix (step 2640).

In general, after completing procedure 2600, the speech recognitionsystem adds an OOV word to its vocabulary to improve future recognitionaccuracy. When the OOV word is added to the vocabulary, the speechrecognition system need not save the adjusted phoneme confusabilitymatrix. However, if the OOV word is not added, the adjusted phonemeconfusability matrix for the OOV word may be saved and accessed in thefuture by the speech recognition system when encountering the OOV wordin a user's utterance.

During alignment (step 2630), the speech recognition system compares asequence of phonemes corresponding to the misrecognized portion of theutterance including the OOV word with the sequence of phonemes for theOOV word. The speech recognition system also generates a list of phonemeconfusions that are likely to be associated with the OOV word. Thespeech recognition system generates this list by determining whichphonemes in the corrected sequence are deleted, inserted, or substitutedto map from the sequence of phonemes for the OOV word to themisrecognized sequence.

Initially, during the recognition, the speech recognition systemattempts to recognize an OOV word or phrase using combinations of words,letters, and phrases in its vocabulary that most closely resemble theOOV word. Such combinations may match the OOV word closely but notnecessarily exactly. For example, if the phrase “recognize” ismisrecognized because “recognize” is not in the vocabulary, the speechrecognition system may substitute the words “wreck a nice” for“recognize” during the recognition. To do this, the speech recognitionsystem substitutes an “s” sound for the “z” sound at the end of the wordand completely drops the “g” sound from the word.

Referring also to FIG. 22, for example, the user has corrected themisrecognized phrase “wreck a nice” with the corrected text “recognize”in a first example 2800. FIG. 22 illustrates two possible alignmentsfrom the substantially larger set of all possible alignments. Ingeneral, the alignments illustrated are more likely to score well thanalignments that are not shown. The first alignment 2805 is shown usingthe solid arrows from the phonemes of “recognize” to the phonemes of“wreck a nice”. In this alignment, the “g” is deleted and “nize” issubstituted with “nice”. The second alignment 2810 is shown using thedotted arrows from the phonemes of “recognize” to the phonemes of “wrecka nice”. In this alignment, the “g” is replaced with “nice” and “nize”is deleted. Scores for each of the alignments are determined using thegeneral phoneme confusability matrix. For example, if the likelihood ofdeleting “g” and substituting “nize” with “nice” is greater than thelikelihood of substituting “g” with “nice” and deleting “nize”, then thespeech recognition system outputs a better score for the first alignment2805 in FIG. 22.

In a more general example 2815, the proofreader has corrected themisrecognized sequence of phonemes “ABDE” with the sequence of phonemes“ABCDE”. In this case, the speech recognition system determines a firstalignment 2820 (shown as solid arrows) as: replace “A” with “A”, replace“B” with “B”, delete “C”, replace “D” with “D”, and replace “E” with“E”. The speech recognition system determines a second alignment 2825(shown as dotted arrows) as: replace “A” with “A”, replace “B” with “B”,replace “C” with “D”, replace “D” with “E”, and delete “E”. Scores foreach of the alignments are determined using the general phonemeconfusability matrix. For example, if the likelihood of deleting “C”,substituting “D” with “D”, and substituting “E” with “E” is greater thanthe likelihood of substituting “C” with “D”, substituting “D” with “E”,and deleting “E”, then the speech recognition system produces a betterscore for the first alignment 2820 in FIG. 22.

Referring again to FIG. 20, the speech recognition system adjusts thecopy of the general phoneme confusability matrix based on one or morebest scoring alignments (step 2635). The speech recognition system makesthis adjustment based on the information about deletion, insertion, orsubstitution obtained from the alignment. For example, the speechrecognition system may adjust the rate or score in the general phonemeconfusability matrix 2700 to reflect a change in the rate ofsubstitution of the phoneme “s” for the phoneme “z.” Thus, the entry forconfusing “z” with “s” has a value of 15 in the general phonemeconfusability matrix 2700. After adjustment, the entry for confusing “z”with “s” may have a value of 1 in an adjusted phoneme confusabilitymatrix, which indicates that “z” is confused with “s” 36% of the timefor this particular OOV word. Although the value may be adjusted to 1 inthis example, the value also may be set empirically. For example, theentry may be changed to 0 for those phonemes that are more confusable.Each time that a particular phoneme confusion is seen in the entries,that number may be used when considering how to adjust the matrix forthat pair of phonemes.

After the speech recognition system has adjusted the general phonemeconfusability matrix (step 2635), the speech recognition system searchesfor the OOV word in the text document using the adjusted matrix (step2640). In general, the search procedure (step 2640) involves searchingfor the phoneme string associated with the OOV word, or likely confusedvariations of the phoneme string, in each utterance in the textdocument. Such a search makes use of the same alignment and scoringprocedures as described above, but now compares the phoneme string forthe OOV word to candidate substrings of each recognized utterance,systematically progressing through the recognized text. If an utterancereceives a score above an empirically-determined threshold (step 2645),the speech recognition system assumes that the utterance includes theOOV word (step 2650) and outputs results (step 2655). Results may beoutput by, for example, highlighting the recognized utterances in thetext document that are likely to include the misrecognized word. In thisway, the proofreader or user may review the highlighted utterances todetermine if further action is needed. Thus, the speech recognitionsystem may present the utterances to the proofreader or user in afashion similar to the highlighting of misspelled words from a spellchecker. Alternatively or in addition, results may be output by, forexample, automatically re-recognizing those utterances that receive ascore above the threshold, now using a vocabulary extended to includethe OOV word.

If an utterance receives a score below or equal to the threshold (step2645), the speech recognition system assumes that the utterance does notinclude the OOV word (step 2660).

Using the procedure 2600, one implementation of the speech recognitionsystem is able to identify approximately 95% of the utterances thatinclude occurrences of the misrecognized word. Moreover, the sameimplementation is able to reject around 95% of the utterances that donot include the misrecognized word. This has resulted in a dramaticimprovement in the proofreading process with respect to correcting OOVwords.

While being applicable primarily to the correction of OOV wordmisrecognitions, the techniques described above also may be applied todetect and correct other recognition errors that are repeated throughouta document. For example, if non-traditional pronunciation by the userresults in the system misrecognizing one vocabulary word for one or moreother vocabulary words, the techniques may be used to detect andhighlight (or even correct) other potential occurrences of the samemisrecognition. It is also important to note that the system does notneed to have produced the same incorrect result for each occurrence of aword in order for those occurrences to be detected. For example, asingle instantiation of the procedure 2600 would detect themisrecognition of “recognize” as both “wreck a nice” and “wreck atnight.” When procedure 2600 is used to correct misrecognitions ofvocabulary words, the speech recognition system would adapt the speechmodels for the user to prevent such misrecognitions from occurring infuture recognitions.

The techniques described above also may be applied to perform efficienttext searching through a large body of speech that has been recognized,a technique referred to as audio mining. For example, when searchingaudio recordings for a unique name (such as the name “Fooberman”), itwould be beneficial to use the above described technique because theunique name may not be in an accessed vocabulary.

Conditional Adaptation

One problem with prior speech recognition systems is the necessity ofobtaining user input to adapt or train the speech models. For example,traditional training techniques fall into one of two distinctcategories: 1) solicitation of user participation in adapting speechmodels and 2) conservative adaptation of speech models. In the firsttechnique, the speech recognition system asks or forces the user tocorrect any mistakes and trains the speech models using the correctedtext. This technique, however, is often tedious to the user because theuser must correct any mistakes. Moreover, this technique is impracticalwhen using a recording device or any sort of mobile speech recognitionsystem because user feedback in that type of system is reduced. In thesecond technique, the speech recognition system rarely adapts the speechmodels, which reduces the time that the user must spend correctingmistakes. However, this technique results in a reduced accuracy in thespeech recognition results because the speech models are adaptedinfrequently. Both of these techniques fail to account for the case whena user is actually changing her mind about the wording and notcorrecting an error in the speech recognition system. When the userchanges her mind, speech models should not be updated.

In a conditional adaptation strategy, the speech recognition systemautomatically determines whether a person is correcting a mistake orchanging her mind during dictation. In one implementation, the user hasdictated a body of text and a speech recognition system has recognizedthe body of text. When the user reads the recognized body of text, theuser may select and edit some text to reflect 1) corrections to amisrecognition and/or 2) revisions to the text that reflect a change ofmind for the user. During the user editing period, the speechrecognition system uses information from the recording of the user'sspeech, the recognition results, and the user's edited text to determinewhether the user is correcting or revising the text.

Referring also to FIG. 23, the speech recognition system performs aprocedure 2900 for adapting acoustic models for a user's speechpatterns. Initially, the speech recognition system receives theuser-dictated text by, for example, receiving a recording from arecorder or receiving user-dictated text directly through a microphone(step 2905).

The speech recognition system then generates the recognized speech (step2910). In one implementation, the speech recognition system recognizesthe user's speech at the same time as adapting acoustic models for theuser's speech patterns. In this case, the user may be editing thedictated text while speaking the dictated text. This may occur when theuser is at a desktop computer dictating text and receiving immediatefeedback from the speech recognition system.

In another implementation, the speech recognition system has alreadyrecognized the user-dictated text or speech and has stored it for lateruse in memory. In this case, the user may have already finished speakingthe dictated text into the mobile recorder or directly into themicrophone and the speech recognition system has stored the dictatedtext into memory.

Next, the speech recognition system receives one or more edits from theuser while the user is reviewing the recognized text (step 2915). Theuser may edit the recognized text using any of the techniques describedabove. For example, the user may edit the recognized text in acorrection dialog or by selecting the text and speaking the correction.

The speech recognition system then determines or builds an acousticmodel for the user-edited text (step 2920). The speech recognitionsystem may determine an acoustic model for the user-edited text bylooking up the text in the vocabulary or in a backup dictionary.Alternatively, if the text is not in the vocabulary or the backupdictionary, the speech recognition system may select acoustic models forthe user-edited text by finding acoustic models that best match theuser-edited text.

As discussed above, an acoustic model may correspond to a word, a phraseor a command from a vocabulary. An acoustic model also may represent asound, or phoneme, which corresponds to a portion of a word.Collectively, the constituent phonemes for a word represent the phoneticspelling of the word. Acoustic models also may represent silence andvarious types of environmental noise.

The speech recognition system calculates an edited acoustic model scorebased on a comparison between the acoustic model for the user-editedtext and acoustic data for the original utterance that the user hadspoken (this acoustic data for the original utterance is stored in thememory) (step 2925). The speech recognition system receives an originalacoustic model score that was determined during recognition and is basedon a comparison between the acoustic model for the recognized utteranceand the acoustic data for the original utterance that the user hadspoken (step 2930). The speech recognition system then calculates adifference between these scores (step 2935) and determines if thisdifference is within a tunable threshold (step 2940) to determinewhether the user-edited text is a correction or a revision to therecognized utterance. If the difference is within a tunable threshold(step 2940), the speech recognition system adapts acoustic models forthe correction of the recognized utterance (step 2945). On the otherhand, if the difference is not within a tunable threshold (step 2940),the speech recognition system does not adapt the acoustic models for therevision to the recognized utterance.

For example, suppose that the user had originally spoken “the cat sat onthe mat” and the speech recognition system recognized this utterance as“the hat sat on the mat”. The acoustic data for the originally spoken“the cat sat on the mat” are stored in memory for future reference. Theuser reviews the recognized text and edits it, in one instance, forcorrection or for revision. If the user decides to correct themisrecognition, the user may select “hat” in the recognized text andspell out the word “cat”. On the other hand, if the user decides torevise the recognition to read “the dog sat on the mat”, then the usermay select “hat” in the recognized text and speak the word “dog”.

When considering the score difference, it is worthwhile to question howthe edited acoustic model score for the user-edited text (called the newscore) could be better than the original acoustic model score for therecognized utterance (called the old score). In this situation, it seemsplausible that the speech recognition system should have detected amistake in the recognized utterance. However, the speech recognitionsystem may fail to detect a mistake. This could occur because the speechrecognition system considers, in addition to the acoustic model, thelanguage model. Another reason for the mistake or oversight could bethat the speech recognition system may have produced a search errorduring recognition, that is, a correct hypothesis could have been prunedduring recognition. One more reason that the speech recognition systemmay fail to detect a mistake may be that the recognized utteranceincluded a new word that was pulled from the backup dictionary.

Another question that arises is how the new score could be a littleworse than the old score. For example, when there is something wrongwith the acoustic model for such a word, the speech recognition systemshould adapt the acoustics or guess a new pronunciation. However, ingeneral, if the new score is much worse than the old score (relative tothe tunable threshold), then the speech recognition system hypothesizesthat the user-edited text corresponds to revisions.

Referring also to FIGS. 24A and 24B, graphs 3000 and 3005 are shown thatmodel the difference between the old score and the new score for eachedited utterance in a sample block of recognition text. The recognitionused a 50,000 word vocabulary. The tester has identified all regions ofthe recognition text that contain errors, and, for each of theseregions, the tester has edited the recognized utterance. The graphs showthe cumulative distribution of these regions, with the score differencebetween the new score and the old score being graphed in a histogram.

In graphs 3000 and 3005, the speech recognition system has performedword recognition on the original user utterance. In graph 3000, thespeech recognition system uses a word vocabulary as a rejection grammar,and in graph 3005, the speech recognition system uses a phoneme sequenceas a rejection grammar.

In graphs 3000 and 3005, the tester has corrected errors in the speechrecognition and results are shown, respectively, in curves 3010 and3015. For example, if the tester originally spoke the utterance “the catsat on the mat” and the speech recognition system incorrectly recognizedthis utterance as “the hat sat on the mat”, the tester may correct therecognition by selecting “hat” and spelling out “cat”. In this case, theold score for the original utterance “cat” would match very nearly thenew score for the recognized utterance “hat” and that is why the speechrecognition system initially made the recognition error. Thus, the scoredifference determined at step 2925 would be relatively small.

Furthermore, the tester has modeled user revisions by making randomedits to the text. Random edits include, in the simplest model, pickingtext at random from the recognition text and deleting that picked text,picking text at random from a choice list and inserting it somewhere inthe recognition text, and picking text at random from the recognitiontext and substituting that picked text with random text from a choicelist.

The random edits are also graphed in histogram form. For graph 3000,these curves are labeled as 3020 (deletion), 3025 (insertion), and 3030(substitution) and for graph 3005, these curves are labeled as 3035(deletion), 3040 (insertion), and 3045 (substitution). Other techniquesfor picking text at random are possible. For example, the tester mayhave picked only function words or only content words.

Using the example in which the user had originally spoken “the cat saton the mat” and the speech recognition system recognized this as “thehat sat on the mat”, the user may, during editing, replace therecognized word “hat” with the word “dog”. In that case, the originalacoustic model score (the old score) is fairly good, while the editedacoustic model score (the new score) is poor because the word “dog”sounds nothing like “cat”, which is what the user originally spoke.Thus, the score difference would be rather large in this case (forexample, 800 on the graph).

On the other hand, if the user replaced the recognized word “hat” withthe word “rat”, then the old score and the new score are both fairlygood. Therefore, the score difference may be relatively small (forexample, 150 on the graph).

Using the above example and graph 3000, if the threshold difference is200 points, then the speech recognition system would adapt on thecorrection from “hat” to “cat”, adapt on the revision from “hat” to“rat”, and ignore the revision from “hat” to “dog”. If the thresholddifference is 100 points, the speech recognition system would adapt onthe correction from “hat” to “cat”, and ignore the revisions from “hat”to “rat” and from “hat” to “dog”.

Evident from the randomly-generated curves 3020-3045 is that they arevery similar to each other in shape and magnitude. Using a threshold of200 difference points, about 5-15% of the randomly-generated revisionsare used by the speech recognition system in adaptation and about 60-95%(where 60% corresponds to phoneme correction and 95% corresponds to wordcorrection) of the corrected text is used by the speech recognitionsystem in adaptation. If the threshold is reduced more, for example, to50 difference points, then many more of the randomly-generated revisionsmay be eliminated from adaptation. However, there will be fewercorrections with which to adapt the speech models.

The techniques and systems described above benefit from the knowledgethat the original recognition results are a fairly good acoustic fit tothe user's speech. Moreover, when language model scores are included,the original recognition results are considered a very good fit to theuser's speech. Additionally, when finding the difference in acousticscores, the scores cancel for those utterances that are unchanged in theedited text and the scores for the corrected or revised utterancesremain to be further analyzed by the speech recognition system. Thus,the techniques and systems may be applied to arbitrarily longutterances, without needing to normalize for the length of theutterance.

Distinguishing Spelling and Dictation During Correction

Referring to FIG. 25, a speech recognition system may be configured todistinguish between correction in the form of spelling correction and inthe form of dictation according to a procedure 3100. When a user selectsmisrecognized text, the user can either speak the pronunciation of thecorrect word or the user can spell the correct word. The speechrecognition system distinguishes between these two correction mechanismswithout requiring the user to indicate which correction mechanism isbeing used. For example, the user need not speak the command “SPELLTHAT” before spelling out the corrected text. As another example, theuser need not speak the command “MAKE THAT” before pronouncing thecorrected text.

Referring also to FIG. 26, a constraint grammar 3200 that permitsspelling and pronunciation in parallel is established (step 3105). Theconstraint grammar 3200 includes a spelling portion in which a firststate 3205 indicates that the first utterance from the user must be aletter in an alphabet and a large vocabulary dictation portion in whicha first state 3210 indicates that the first utterance from the user mustbe a word from the dictation vocabulary. A path 3215, which returns tothe first state 3205, indicates that the utterance may includeadditional letters. A path 3220, which exits the first state 3205 andcompletes the utterance, indicates that the utterance may include onlyletters. A path 3225, which returns to the second state 3210, indicatesthat the utterance may include additional words from the dictationvocabulary. A path 3230, which exits the second state 3210 and completesthe utterance, indicates that the utterance may include only words fromthe dictation vocabulary.

The large vocabulary dictation portion also indicates the frequency withwhich words occur. For example, a language model associated with thelarge vocabulary dictation portion may be a unigram model that indicatesthe frequency with which a word occurs independently of context or abigram model that indicates the frequency with which a word occurs inthe context of a preceding word. For example, a bigram model mayindicate that a noun or adjective is more likely to follow the word“the” than a verb or preposition.

Similarly, the spelling portion may indicate the frequency with whichletters occur. For example, a language model associated with thespelling portion may be a unigram model that indicates the frequencywith which a letter occurs independently of context or a bigram modelthat indicates the frequency with which a letter occurs in the contextof a preceding letter. For example, a bigram model may indicate that avowel is more likely to follow the letter “m” than a consonant.

Referring again to FIG. 25, a fixed biasing value between the spellingand dictation grammars may be introduced to improve the chances that thespeech recognition system distinguishes a spelled correction from apronounced correction (step 3110).

After the constraint grammar is established (step 3105), the constraintgrammar may be implemented during error correction (step 3115). In thismanner, during correction, the speech recognition system initiallydetermines if the user is correcting an error. If so, the systemrecognizes the user's correction using the established constraintgrammar. If the user corrects the misrecognition by spelling out thecorrect word, the speech recognition system determines that thecorrection follows the path through the state 3205 and determines thecorrection accordingly. If the user corrects the misrecognition bypronouncing the correct word, the speech recognition system determinesthat the correction follows the path through the state 3210 anddetermines the correction accordingly.

Because both of the constraint grammar portions are used in parallel bythe speech recognition system, the system is able to determine whichportion gives the most likely recognition result.

If a fixed biasing value is introduced between the spelling portion andthe dictation portion, then the speech recognition system considers thebiasing when selecting between the portions. For example, the biasingvalue may indicate that the user is more likely to dictate a correctionthan to spell it, such that the score for spelling portions will need tobe better than that of the dictation portion by more than the biasingvalue in order to be selected.

The techniques described here are not limited to any particular hardwareor software configuration; they may find applicability in any computingor processing environment that may be used for speech recognition. Thetechniques may be implemented in hardware or software, or a combinationof the two. Preferably, the techniques are implemented in computerprograms executing on programmable computers that each include aprocessor, a storage medium readable by the processor (includingvolatile and non-volatile memory and/or storage elements), at least oneinput device, and at least one output device. Program code is applied todata entered using the input device to perform the functions describedand to generate output information. The output information is applied toone or more output devices.

Each program is preferably implemented in a high level procedural orobject oriented programming language to communicate with a computersystem. However, the programs can be implemented in assembly or machinelanguage, if desired. In any case, the language may be a compiled orinterpreted language.

Each such computer program is preferably stored on a storage medium ordevice (for example, CD-ROM, hard disk or magnetic diskette) that isreadable by a general or special purpose programmable computer forconfiguring and operating the computer when the storage medium or deviceis read by the computer to perform the procedures described in thisdocument. The system may also be considered to be implemented as acomputer-readable storage medium, configured with a computer program,where the storage medium so configured causes a computer to operate in aspecific and predefined manner.

1-32. (canceled)
 33. A computer-implemented method for speechrecognition, the method comprising: receiving dictated text; generatingrecognized speech based on the received dictated text, the generatingcomprising determining acoustic models for the dictated text that bestmatch acoustic data for the dictated text; receiving an edited text ofthe recognized speech, the edited text indicating a replacement for aportion of the dictated text; determining an acoustic model for theedited text; determining whether to adapt acoustic models for the editedtext based on the acoustic model for the edited text and the acousticmodel for the dictated text portion.
 34. The method of claim 33 furthercomprising calculating an acoustic model score based on a comparisonbetween the acoustic model for the edited text and the acoustic data forthe dictated text portion.
 35. The method of claim 34 in whichdetermining whether to adapt acoustic models for the edited text isbased on the calculated acoustic model score.
 36. The method of claim 35in which determining whether to adapt acoustic models for the editedtext comprises calculating an original acoustic model score based on acomparison between the acoustic model for the dictated text portion andthe acoustic data for the dictated text portion.
 37. The method of claim36 in which determining whether to adapt acoustic models for the editedtext comprises calculating a difference between the acoustic model scoreand the original acoustic model score.
 38. The method of claim 37 inwhich determining whether to adapt acoustic models for the edited textcomprises determining whether the difference is less than apredetermined value.
 39. The method of claim 38 in which determiningwhether to adapt acoustic models for the edited text comprises adaptingacoustic models for the edited text if the difference is less than apredetermined value.
 40. The method of claim 38 in which determiningwhether to adapt acoustic models for the edited text comprises bypassingadapting acoustic models for the edited text if the difference isgreater than or equal to a predetermined value.
 41. The method of claim33 in which receiving the edited text of the recognized speech occursduring a recognition session in which the recognized speech isgenerated.
 42. The method of claim 33 in which receiving the edited textof the recognized speech occurs after a recognition session in which therecognized speech is generated.
 43. The method of claim 33 in whichreceiving the edited text of the recognized speech comprises receiving aselection of the portion of the dictated text.
 44. The method of claim33 in which determining an acoustic model for the edited text comprisessearching for the edited text in a vocabulary or a backup dictionaryused to generate the recognized speech.
 45. The method of claim 33 inwhich determining an acoustic model for the edited text comprisesselecting an acoustic model that best matches the edited text.
 46. Acomputer-implemented method of speech recognition, the methodcomprising: performing speech recognition on an utterance to produce arecognition result for the utterance; receiving a selection of therecognition result; receiving a correction of the recognition result;performing speech recognition on the correction using a constraintgrammar that permits spelling and pronunciation in parallel; andidentifying whether the correction comprises a spelling or apronunciation using the constraint grammar.
 47. The method of claim 46further comprising generating a replacement result for the recognitionresult based on the correction.
 48. The method of claim 46 in which theconstraint grammar includes a spelling portion and a dictationvocabulary portion.
 49. The method of claim 48 in which the spellingportion indicates that the first utterance from the user is a letter inan alphabet.
 50. The method of claim 48 in which the vocabulary portionindicates that the first utterance from the user is a word from thedictation vocabulary.
 51. The method of claim 48 in which the spellingportion indicates a frequency with which letters occur in a languagemodel.
 52. The method of claim 48 in which the dictation vocabularyportion indicates a frequency with which words occur in a languagemodel.
 53. The method of claim 48 further comprising introducing abiasing value between the spelling and the dictation vocabulary portionsof the constraint grammar.